Deep understanding of shopper behaviours and interactions using RGB-D vision

AbstractIn retail environments, understanding how shoppers move about in a store's spaces and interact with products is very valuable. While the retail environment has several favourable characteristics that support computer vision, such as reasonable lighting, the large number and diversity of products sold, as well as the potential ambiguity of shoppers' movements, mean that accurately measuring shopper behaviour is still challenging. Over the past years, machine-learning and feature-based tools for people counting as well as interactions analytic and re-identification were developed with the aim of learning shopper skills based on occlusion-free RGB-D cameras in a top-view configuration. However, after moving into the era of multimedia big data, machine-learning approaches evolved into deep learning approaches, which are a more powerful and efficient way of dealing with the complexities of human behaviour. In this paper, a novel VRAI deep learning application that uses three convolutional neural networks to count the number of people passing or stopping in the camera area, perform top-view re-identification and measure shopper–shelf interactions from a single RGB-D video flow with near real-time performances has been introduced. The framework is evaluated on the following three new datasets that are publicly available: TVHeads for people counting, HaDa for shopper–shelf interactions and TVPR2 for people re-identification. The experimental results show that the proposed methods significantly outperform all competitive state-of-the-art methods (accuracy of 99.5% on people counting, 92.6% on interaction classification and 74.5% on re-id), bringing to different and significative insights for implicit and extensive shopper behaviour analysis for marketing applications

Guideline for Care of Patients with the Diagnoses of Craniosynostosis: Working Group on Craniosynostosis

This guideline for care of children with craniosynostosis was developed by a national working group with representatives of 11 matrix societies of specialties and the national patients' society. All medical aspects of care for nonsyndromic and syndromic craniosynostosis are included, as well as the social and psychologic impact for the patient and their parents. Managerial aspects are incorporated as well, such as organizing a timely referral to the craniofacial center, requirements for a dedicated craniofacial center, and centralization of this specialized care. The conclusions and recommendations within this document are founded on the available literature, with a grading of the level of evidence, thereby highlighting the areas of care that are in need of high-quality research. The development of this guideline was made possible by an educational grant of the Dutch Order of Medical Specialists. The development of this guideline was supported by an educational grant of the Dutch Order of Medical Specialists

Deep Understanding of Shopper Behaviours and Interactions in Intelligent Retail Environment

In ambienti retail comprendere come il consumatore si muove nello spazio e interagisce con i prodotti risulta essere di notevole interesse. Nonostante l'ambiente retail possegga diverse caratteristiche favorevoli al supporto della computer vision, ad esempio un'illuminazone costante, il vasto numero e la variabilità dei prodotti venduti, così come la potenziale ambiguità dei movimenti del comsumatore, indicano che misurarne il comportamento è tuttora sfidante. Negli anni, tecniche di machine learning e feature-based per il conteggio persone, l'analisi delle interazioni e la re-identificazione sono state sviluppate allo scopo di apprendere il comportamento del consumatore, basandosi su camere RGB-D in configurazione top-view. Tuttavia dall'avvento dei big data gli approcci machine learning sono evoluti verso approcci deep learning, che risultano essere un mezzo più potente ed efficiente per trattare la complessità del comportamento umano. Partendo da questa premessa questa tesi tratta l'evoluzione di 3 sistemi reali quali: People Counting, Shopper Analytics e Re-Identification. L'obbiettivo principale è quello di sviluppare architetture deep learning progettate specificatamente per ambito retail. A questo scopo un nuovo VRAI deep learning framework viene descritto. In particolare utilizza 3 reti neurali convoluzionali (CNN) per contare il numero di persone che passano o si fermano nell'area coperta dalla camera, effettuare una re-identificazione top-view e misurare le interazioni consumatore-scaffale da un singolo flusso RGBD con performance quasi real-time. Il VRAI framework è stato poi valutato su 3 nuovi dataset resi pubblici: TVHeads per il conteggio persone, HaDa per l'analisi delle interazioni consumatore-scaffale e TVPR2 per la re-identificazione.In retail environments, understanding how shoppers move in the store’s spaces and interact with products is very valuable. While the retail environment has several favourable characteristics that support computer vision, such as reasonable lighting, the large number and diversity of products sold, as well as the potential ambiguity of shoppers’ movements, mean that accurately measuring shopper behaviour is still challenging. Over the past years, machine-learning and feature-based tools for people counting as well as interactions analytics and re-identification were developed with the aim of learning shopper behaviors based on occlusion-free RGB-D cameras in a top-view configuration. However,after moving into the era of multimedia big data, machine-learning approaches evolved into deep learning approaches, which are a more powerful and efficient way of dealing with the complexities of human behaviour. Starting from such a premise, this thesis addresses the evolution process of 3 real systems such as: People Counting, Shopper Analytics and Re-Identification. The main goal is to develop Deep Learning architectures especially designed for Retail Environment. For this purpose, a novel VRAI deep learning framework is described. In particular, it uses 3 Convolutional Neural Networks (CNNs) to count the number of people passing or stopping in the camera area, perform top-view re-identification and measure shopper-shelf interactions from a single RGB-D video flow with near real-time performances. The VRAI framework is evaluated on the following 3 new datasets that are publicly available: TVHeads for people counting, HaDa for shopper-shelf interactions and TVPR2 for people re-identification

Preterm infants’ limb-pose estimation from depth images using convolutional neural networks

Preterm infants' limb-pose estimation is a crucial but challenging task, which may improve patients' care and facilitate clinicians in infant's movements monitoring. Work in the literature either provides approaches to whole-body segmentation and tracking, which, however, has poor clinical value, or retrieve a posteriori limb pose from limb segmentation, increasing computational costs and introducing inaccuracy sources. In this paper, we address the problem of limb-pose estimation under a different point of view. We proposed a 2D fully-convolutional neural network for roughly detecting limb joints and joint connections, followed by a regression convolutional neural network for accurate joint and joint-connection position estimation. Joints from the same limb are then connected with a maximum bipartite matching approach. Our analysis does not require any prior modeling of infants' body structure, neither any manual interventions. For developing and testing the proposed approach, we built a dataset of four videos (video length = 90 s) recorded with a depth sensor in a neonatal intensive care unit (NICU) during the actual clinical practice, achieving median root mean square distance [pixels] of 10.790 (right arm), 10.542 (left arm), 8.294 (right leg), 11.270 (left leg) with respect to the ground-truth limb pose. The idea of estimating limb pose directly from depth images may represent a future paradigm for addressing the problem of preterm-infants' movement monitoring and offer all possible support to clinicians in NICUs

Modelling and Forecasting Customer Navigation in Intelligent Retail Environments

Understanding shopper behaviour is one of the keys to success for retailers. In particular, it is necessary that managers know which retail attributes are important to which shoppers and their main goal is to improve the consumer shopping experience. In this work, we present sCREEN (Consumer REtail ExperieNce), an intelligent mechatronic system for indoor navigation assistance in retail environments that minimizes the need for active tagging and does not require metrics maps. The tracking system is based on Ultra-wideband technology. The digital devices are installed in the shopping carts and baskets and sCREEN allows modelling and forecasting customer navigation in retail environments. This paper contributes the design of an intelligent mechatronic system with the use of a novel Hidden Markov Models (HMMs) for the representation of shoppers\u2019 shelf/category attraction and usual retail scenarios such as product out of stock or changes on store layout. Observations are viewed as a perceived intelligent system performance. By forecasting consumers next shelf/category attraction, the system can present the item location information to the consumer, including a walking route map to a location of the product in the retail store, and/or the number of an aisle in which the product is located. Effective and efficient design processes for mechatronic systems are a prerequisite for competitiveness in an intelligent retail environment. Experiments are performed in a real retail environment that is a German supermarket, during business hours. A dataset, with consumers trajectories, timestamps and the corresponding ground truth for training as well as evaluating the HMM, have been built and made publicly available. The results in terms of Precision, Recall and F1-score demonstrate the effectiveness and suitability of our approach, with a precision value that exceeds the 76% in all test cases

Convolutional Networks for Semantic Heads Segmentation using Top-View Depth Data in Crowded Environment

Detecting and tracking people is a challenging task in a persistent crowded environment (i.e. retail, airport, station, etc.) for human behaviour analysis of security purposes. This paper introduces an approach to track and detect people in cases of heavy occlusions based on CNNs for semantic segmentation using top-view depth visual data. The purpose is the design of a novel U-Net architecture, U-Net3, that has been modified compared to the previous ones at the end of each layer. In particular, a batch normalization is added after the first ReLU activation function and after each max-pooling and up-sampling functions. The approach was applied and tested on a new and public available dataset, TVHeads Dataset, consisting of depth images of people recorded from an RGB-D camera installed in top-view configuration. Our variant outperforms baseline architectures while remaining computationally efficient at inference time. Results show high accuracy, demonstrating the effectiveness and suitability of our approach

Generating depth images of preterm infants in given poses using GANs

The use of deep learning for preterm infant's movement monitoring has the potential to support clinicians in early recognizing motor and behavioural disorders. The development of deep learning algorithms is, however, hampered by the lack of publicly available annotated datasets